fix(quantization): handle mixed-precision fused layers in is_layer_skipped_gguf - #107
Open
BenWongCityuCS wants to merge 1 commit into
Open
fix(quantization): handle mixed-precision fused layers in is_layer_skipped_gguf#107BenWongCityuCS wants to merge 1 commit into
BenWongCityuCS wants to merge 1 commit into
Conversation
…ipped_gguf
This fixes two hard failures when loading mixed-precision ("UD"/"XL"-style)
GGUF on vLLM 0.27 (e.g. `unsloth/Kimi-K3-GGUF` `UD-Q2_K_XL`):
1. `is_layer_skipped_gguf` raised `ValueError: Detected some but not all
shards of ... are quantized` for fused layers whose shards are at
different precisions — UD files are mixed by design (e.g. KDA
`self_attn.in_proj_qkvgfab` in vLLM 0.27's Kimi-K3).
2. If such a fused layer were routed to the quantized path its only
parameter is `qweight`, but `KimiK3DeltaAttention.__init__` writes
`in_proj_qkvgfab.weight.data[...].zero_()` → `AttributeError: ... has no
attribute 'weight'`.
Fix: for heterogeneous fused shards, keep `in_proj_qkvgfab` UNQUANTIZED so a
real `.weight` exists and the model's own replicated-shard padding zeroing
runs, instead of aborting. Other mixed fused layers remain on the quantized
path.
Tested end-to-end on vLLM 0.27.0 (`vllm-openai:v0.27.0-cu129`), 8x141GB H20,
TP8 + `--enable-expert-parallel`, serving `UD-Q2_K_XL` with `--quantization
gguf` (text-only Kimi-K3 arch); boots and answers normally.
Related: vllm-project#92 (Kimi-K3 GGUF adapter). Informational note for vllm-project#92, not required
here: `GGUFMoEMethod.apply` currently discards `shared_experts`/`shared_experits_input`,
and Kimi-K3 has `expert_shared_count=2` — if those are wired through the
fused-MoE path they would not be computed.
Signed-off-by: Ben Wong <116333430+BenWongCityuCS@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes two hard failures when loading mixed-precision ("UD"/"XL"-style) GGUF on vLLM 0.27 (e.g.
unsloth/Kimi-K3-GGUFUD-Q2_K_XL):is_layer_skipped_ggufraisedValueError: Detected some but not all shards of ... are quantizedfor fused layers whose shards are at different precisions — UD files are mixed by design (e.g. KDAself_attn.in_proj_qkvgfabin vLLM 0.27's Kimi-K3).qweight, butKimiK3DeltaAttention.__init__writesin_proj_qkvgfab.weight.data[...].zero_()→AttributeError: ... has no attribute 'weight'.Fix: for heterogeneous fused shards, keep
in_proj_qkvgfabUNQUANTIZED so a real.weightexists and the model's own replicated-shard padding zeroing runs, instead of aborting. Other mixed fused layers remain on the quantized path.Tested end-to-end on vLLM 0.27.0 (
vllm-openai:v0.27.0-cu129), 8x141GB H20, TP8 +--enable-expert-parallel, servingUD-Q2_K_XLwith--quantization gguf(text-only Kimi-K3 arch); boots and answers normally.docker run -d
--name vllm-k3gguf-serve
--gpus '"device=0,1,2,3,4,5,6,7"'
--ipc=host
--ulimit memlock=-1
--ulimit stack=67108864
-v /models:/ssd1
-p 31001:31001
--entrypoint python3
-e VLLM_USE_RUST_FRONTEND=1
vllm/vllm-openai:v0.27.0-k3gguf-mxf2
-m vllm.entrypoints.openai.api_server
--quantization gguf
--model /ssd1/Kimi-K3-GGUF/UD-Q2_K_XL/Kimi-K3-UD-Q2_K_XL-00001-of-00019.gguf
--tokenizer /ssd1/Kimi-K3-GGUF/UD-Q2_K_XL/
--served-model-name Kimi-K3-GGUF
--max-num-seqs 5
--max-model-len 262144
--max-num-batched-tokens 2048
--enable-prefix-caching
--host 0.0.0.0
--port 31001
--trust-remote-code
--enable-expert-parallel
--tensor-parallel-size 8
--gpu-memory-utilization 0.95
--no-enable-flashinfer-autotune
--enable-auto-tool-choice
--tool-call-parser kimi_k3
--reasoning-parser kimi_k3
--disable-uvicorn-access-log
Related: #92 (Kimi-K3 GGUF adapter).